DataFrame Schemas

A schema defines the column names and data types of a DataFrame. While Spark can infer schemas automatically from structured sources (like Parquet or JSON), defining explicit schemas is a production best practice for building robust, reliable data pipelines.

classDiagram
    class StructType {
        +List~StructField~ fields
    }
    class StructField {
        +String name
        +DataType dataType
        +Boolean nullable
    }
    StructType --> StructField : contains multiple columns

Schema Inference vs. Explicit Schema

When you load data without specifying a schema, Spark performs a complete scan over the dataset to guess the data type of each column.

Defining Programmatic Schemas in PySpark

To define an explicit schema, PySpark provides the pyspark.sql.types module, specifically:

• StructType: Represents a collection of fields (a row).
• StructField: Represents a single column with a name, data type, and boolean indicating nullability.

Code Example: Creating a Schema Programmatically

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, DoubleType

# 1. Initialize Spark
spark = SparkSession.builder \
    .appName("DataFrame Schemas") \
    .master("local[*]") \
    .getOrCreate()

# 2. Define the schema programmatically
schema = StructType([
    StructField("employee_id", IntegerType(), nullable=False),
    StructField("first_name", StringType(), nullable=True),
    StructField("last_name", StringType(), nullable=True),
    StructField("salary", DoubleType(), nullable=True),
    StructField("department", StringType(), nullable=True)
])

# 3. Dummy dataset matching the schema
data = [
    (101, "Alice", "Smith", 85000.0, "Engineering"),
    (102, "Bob", "Jones", 72000.0, "Marketing"),
    (103, "Charlie", "Brown", 91000.0, "Engineering")
]

# 4. Create DataFrame enforcing the schema
df = spark.createDataFrame(data, schema=schema)

# 5. Inspect the schema structure and print types
df.printSchema()
df.show()

Schema Console Output

When you call printSchema(), Spark outputs a clean tree:

root
 |-- employee_id: integer (nullable = false)
 |-- first_name: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- salary: double (nullable = true)
 |-- department: string (nullable = true)

Enforcing Schemas on File Ingestion

When reading unstructured files like CSV, enforcing your schema ensures Spark doesn't have to read the file twice:

# Ingesting CSV file while enforcing explicit schema programmatically
csv_df = spark.read \
    .format("csv") \
    .option("header", "true") \
    .schema(schema) \
    .load("dataset.csv")